Practical AI on the Google Cloud Platform by Micheal Lanham

Author:Micheal Lanham [Micheal Lanham]
Language: eng
Format: epub
Publisher: O'Reilly Media, Inc.
Published: 2020-12-24T16:00:00+00:00

SGD and Batching Samples

One problem we come across when training thousands of data through a network is it can take a long time and isn’t general enough. That is, if we update our network for each individual weight we may find elements that cancel each other out. This can further be compounded if the data is pulled from the same order. In order to alleviate these problems we introduce a random batching approach to updating our network. By batching the data into groups of data and then applying changes averaged across those groups better generalizes the network, which is generally a good thing. Furthermore, we randomize this batching process so that no 2 batches are alike and data is further processed randomly. This whole technique is called stochastic gradient descent when used with backpropagation in order to train a deep learning network.

We use the term stochastic to mean random, since we are now pulling random groups of samples. The gradient descent part is the heart of backpropagation optimization as we already learned. Stochastic Gradient Descent or SGD is the standard optimizer as we’ve seen earlier. There are plenty of variations to SGD that are more powerful and we will explore those as well. The important thing to remember about SGD and other optimizers is that they use batches of data and no longer individual samples. As it turns out, since we are using linear systems of equations this also becomes more computationally efficient.

batch_size = the batch size used to determine updates to the network. Typical batch sizes are 32 - 256 for large dataset sizes. The batch size is a deceptive parameter that may or may not have an incredible impact on network training. It generally will be one of the first parameters you tune in enhance a network.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.